Spatio-temporal Feature Extraction Research Articles

Adaptive temporal compression for reduction of computational complexity in human behavior recognition

The research on video analytics especially in the area of human behavior recognition has become increasingly popular recently. It is widely applied in virtual reality, video surveillance, and video retrieval. With the advancement of deep learning algorithms and computer hardware, the conventional two-dimensional convolution technique for training video models has been replaced by three-dimensional convolution, which enables the extraction of spatio-temporal features. Specifically, the use of 3D convolution in human behavior recognition has been the subject of growing interest. However, the increased dimensionality has led to challenges such as the dramatic increase in the number of parameters, increased time complexity, and a strong dependence on GPUs for effective spatio-temporal feature extraction. The training speed can be considerably slow without the support of powerful GPU hardware. To address these issues, this study proposes an Adaptive Time Compression (ATC) module. Functioning as an independent component, ATC can be seamlessly integrated into existing architectures and achieves data compression by eliminating redundant frames within video data. The ATC module effectively reduces GPU computing load and time complexity with negligible loss of accuracy, thereby facilitating real-time human behavior recognition.

An IIoT Temporal Data Anomaly Detection Method Combining Transformer and Adversarial Training

The existing Industrial Internet of Things (IIoT) temporal data analysis methods often suffer from issues such as information loss, difficulty balancing spatial and temporal features, and being affected by training data noise, which can lead to varying degrees of reduced model accuracy. Therefore, a new anomaly detection method was proposed, which integrated Transformer and adversarial training. Firstly, a bidirectional spatiotemporal feature extraction module was constructed by combining Graph Attention Networks (GAT) and Bidirectional Gated Recurrent Unit (BiGRU), which can simultaneously extract spatial and temporal features. Then, by combining multi-scale convolution with Long Short-Term Memory (LSTM), multi-scale contextual information was captured. Finally, an improved Transformer was used to fuse multi-dimensional features, combined with an adversarial-trained variational autoencoder to calculate the anomalies of the input data. This method outperforms other comparison models by conducting experiments on four publicly available datasets.

Reservoir fluid identification based on multi-head attention with UMAP

Gas logging curves obtained during drilling, along with conventional well logs, typically serve as foundational data for identifying reservoir fluids. Traditional methods for identifying layers containing oil or gas often suffer from limitations such as inadequate feature extraction, underutilization of differences in data points, and limited applicability. These limitations lead to reduced accuracy in predicting reservoir fluid segmentation. To address the challenge of spatiotemporal feature extraction in logging curves, this paper presents a fluid identification method based on deep learning networks and multivariate sequence classification. The method takes into account both the sequential variations in geological strata and the relationships among different curves for fluid prediction. Our proposed method utilizes a Transformer multi-head attention mechanism and follows a series of steps. First, an improved Symbolic Aggregate approXimation (SAX) is employed to assess the response characteristics of the logging curves to the target fluid and identify sensitive curves based on information similarity. Subsequently, the semi-supervised learning method UMAP is applied to identify layers with high interpretability. Hierarchical training is then employed to mitigate the impact of noise and high-dimensional nonlinearity on identification results. Finally, the processed vector curves are input into the attention module layer, where features of each layer are extracted in an encoding-decoding manner. This method has been tested and validated using gas logging data from 12 wells in a gas field offshore China. The overall prediction accuracy can exceed 94%, and the prediction accuracy for a single gas layer is over 88%. When compared to other machine learning approaches, such as K-means, decision tree, and XGBoost, this method exhibits superior identification accuracy and generalization ability, especially for unbalanced samples.

RadarTCN: Lightweight Online Classification Network for Automotive Radar Targets Based on TCN.

Automotive radar is one of the key sensors for intelligent driving. Radar image sequences contain abundant spatial and temporal information, enabling target classification. For existing radar spatiotemporal classifiers, multi-view radar images are usually employed to enhance the information of the target and 3D convolution is employed for spatiotemporal feature extraction. These models consume significant hardware resources and are not applicable to real-time applications. In this paper, RadarTCN, a novel lightweight network, is proposed that achieves high-accuracy online target classification using single-view radar image sequences only. In RadarTCN, 2D convolution and 3D-TCN are employed to extract spatiotemporal features sequentially. To reduce data dimensionality and computational complexity, a multi-layer max pooling down-sampling method is designed in a 2D convolution module. Meanwhile, the 3D-TCN module is improved through residual pruning and causal convolution is introduced for leveraging the performance of online target classification. The experimental results demonstrate that RadarTCN can achieve high-precision online target recognition for both range-angle and range-Doppler map sequences. Compared to the reference models on the CARRADA dataset, RadarTCN exhibits better classification performance, with fewer parameters and lower computational complexity.

[Automatic epilepsy detection with an attention-based multiscale residual network].

The deep learning-based automatic detection of epilepsy electroencephalogram (EEG), which can avoid the artificial influence, has attracted much attention, and its effectiveness mainly depends on the deep neural network model. In this paper, an attention-based multi-scale residual network (AMSRN) was proposed in consideration of the multiscale, spatio-temporal characteristics of epilepsy EEG and the information flow among channels, and it was combined with multiscale principal component analysis (MSPCA) to realize the automatic epilepsy detection. Firstly, MSPCA was used for noise reduction and feature enhancement of original epilepsy EEG. Then, we designed the structure and parameters of AMSRN. Among them, the attention module (AM), multiscale convolutional module (MCM), spatio-temporal feature extraction module (STFEM) and classification module (CM) were applied successively to signal reexpression with attention weighted mechanism as well as extraction, fusion and classification for multiscale and spatio-temporal features. Based on the Children's Hospital Boston-Massachusetts Institute of Technology (CHB-MIT) public dataset, the AMSRN model achieved good results in sensitivity (98.56%), F1 score (98.35%), accuracy (98.41%) and precision (98.43%). The results show that AMSRN can make good use of brain network information flow caused by seizures to enhance the difference among channels, and effectively capture the multiscale and spatio-temporal features of EEG to improve the performance of epilepsy detection.

Video anomaly detection based on attention and efficient spatio-temporal feature extraction

Video anomaly detection based on attention and efficient spatio-temporal feature extraction

An estimation method for switching points of multimode spatiotemporal data based on SFSTAR

Various traditional methods, including statistical and spatial map-based approaches, are employed to model spatiotemporal data. However, the lack of explicit descriptions of spatial and temporal dependencies leads to unnecessarily intensive and uninterpretable relationships among the variables. To address this problem, space–time autoregressive moving average (STARMA)-type models that can depict temporal and spatial dependencies in a parametric manner are adopted in this study. However, such models may not be applicable to unlabeled multimode spatiotemporal data. In this study, we propose a switching flexible space–time autoregressive (SFSTAR) model to describe multimode spatiotemporal data. Considering the coupling between the parameter estimation of each mode and determination of the switching points, an estimation method is proposed to identify the mode switching points to partition the multimode data into different segments. First, the proposed estimation method identifies the number of segments by locating the transition regions that enclose the switching points based on the spatiotemporal feature extraction proposed herein. Second, with the estimated number of segments that is actually the number modes and the constraints on the locations of the switching points given by the transition regions, the proposed method jointly estimates the switching points and the SFSTAR model parameters in each mode through Bayesian optimization. The effectiveness of the proposed method is evaluated using a numerical example and a fiber optic temperature measurement dataset.

Network Traffic Classification Model Based on Spatio-Temporal Feature Extraction

The demand for encrypted communication is increasing with the continuous development of secure and trustworthy networks. In edge computing scenarios, the requirement for data processing security is becoming increasingly high. Therefore, the accurate identification of encrypted traffic has become a prerequisite to ensure edge intelligent device security. Currently, encrypted network traffic classification relies on single-feature extraction methods. These methods have simple feature extraction, making distinguishing encrypted network data flows and designing compelling manual features challenging. This leads to low accuracy in multi-classification tasks involving encrypted network traffic. This paper proposes a hybrid deep learning model for multi-classification tasks to address this issue based on the synergy of dilated convolution and gating unit mechanisms. The model comprises a Gated Dilated Convolution (GDC) module and a CA-LSTM module. The GDC module completes the spatial feature extraction of encrypted network traffic through dilated convolution and gating unit mechanisms. In contrast, the CA-LSTM module focuses on extracting temporal network traffic features. By employing a collaborative approach to extract spatio-temporal features, the model ensures feature extraction diversity, guarantees robustness, and effectively enhances the feature extraction rate. We evaluate our multi-classification model using the ISCX VPN-nonVPN public dataset. Experimental results show that the proposed method achieves an accuracy rate of over 95% and a recall rate of over 90%, significantly outperforming existing methods.

Spatiotemporal crowds features extraction of infrared images using neural network

Spatiotemporal crowds features extraction of infrared images using neural network

End-to-End Network Intrusion Detection Based on Contrastive Learning.

The network intrusion detection system (NIDS) plays a crucial role as a security measure in addressing the increasing number of network threats. The majority of current research relies on feature-ready datasets that heavily depend on feature engineering. Conversely, the increasing complexity of network traffic and the ongoing evolution of attack techniques lead to a diminishing distinction between benign and malicious network behaviors. In this paper, we propose a novel end-to-end intrusion detection framework based on a contrastive learning approach. We design a hierarchical Convolutional Neural Network (CNN) and Gated Recurrent Unit (GRU) model to facilitate the automated extraction of spatiotemporal features from raw traffic data. The integration of contrastive learning amplifies the distinction between benign and malicious network traffic in the representation space. The proposed method exhibits enhanced detection capabilities for unknown attacks in comparison to the approaches trained using the cross-entropy loss function. Experiments are carried out on the public datasets CIC-IDS2017 and CSE-CIC-IDS2018, demonstrating that our method can attain a detection accuracy of 99.9% for known attacks, thus achieving state-of-the-art performance. For unknown attacks, a weighted recall rate of 95% can be achieved.

Finding Visual Saliency in Continuous Spike Stream

As a bio-inspired vision sensor, the spike camera emulates the operational principles of the fovea, a compact retinal region, by employing spike discharges to encode the accumulation of per-pixel luminance intensity. Leveraging its high temporal resolution and bio-inspired neuromorphic design, the spike camera holds significant promise for advancing computer vision applications. Saliency detection mimic the behavior of human beings and capture the most salient region from the scenes. In this paper, we investigate the visual saliency in the continuous spike stream for the first time. To effectively process the binary spike stream, we propose a Recurrent Spiking Transformer (RST) framework, which is based on a full spiking neural network. Our framework enables the extraction of spatio-temporal features from the continuous spatio-temporal spike stream while maintaining low power consumption. To facilitate the training and validation of our proposed model, we build a comprehensive real-world spike-based visual saliency dataset, enriched with numerous light conditions. Extensive experiments demonstrate the superior performance of our Recurrent Spiking Transformer framework in comparison to other spike neural network-based methods. Our framework exhibits a substantial margin of improvement in capturing and highlighting visual saliency in the spike stream, which not only provides a new perspective for spike-based saliency segmentation but also shows a new paradigm for full SNN-based transformer models. The code and dataset are available at https://github.com/BIT-Vision/SVS.

Spatio-temporal multivariable time vario-zoom network for water level forecasting based on high-resolution hydrological dataset

Water level forecasting is a crucial element in river-basin water resource management, particularly for shipping and extreme events, such as floods and droughts. However, accurately forecasting water levels is challenging due to the influence of non-stationarity hydrological processes, complex characteristic variables, and noisy data. Data-driven approaches in water level forecasting generally lack annotated high-resolution data. To address these problems, this paper presents the first hydrological dataset on the Xijiang River basin-Xijiang Water Dataset (XJWD) and a Spatio-Temporal Multivariable-based Time Vario-Zoom Network (STM- TVZN) model. XJWD combines multi-source raw hydrological data and has real-time and higher resolution data than comparable datasets. STM-TVZN is specifically tailored to enhance the discernment of salient features within key time windows for each variable by fusing spatio-temporal multivariate features through vario-zoom time windows. Finally, a novel metric, Extremum Time Offsets (ETO), is proposed to evaluate the accuracy of the occurrence of local extremum in time series, and it is significant for predicting and evaluating extremum events such as floods and droughts. Experimental results show that the proposed method outperformed existing models on XJWD dataset regarding error rate, stationarity, similarity, and ETO. Overall, the proposed method of combining time-variable focusing network design and spatio-temporal multivariate feature extraction is appealing for short-term water level forecasting in inland rivers, exhibiting potential capacity in strategic water-saving planning and extremum event forecasting caused by climate change.

VST-PCA: A Land Use Change Simulation Model Based on Spatiotemporal Feature Extraction and Pre-Allocation Strategy

Land use/cover change (LUCC) refers to the phenomenon of changes in the Earth’s surface over time. Accurate prediction of LUCC is crucial for guiding policy formulation and resource management, contributing to the sustainable use of land, and maintaining the health of the Earth’s ecosystems. LUCC is a dynamic geographical process involving complex spatiotemporal dependencies. Existing LUCC simulation models suffer from insufficient spatiotemporal feature learning, and traditional cellular automaton (CA) models exhibit limitations in neighborhood effects. This study proposes a cellular automaton model based on spatiotemporal feature learning and hotspot area pre-allocation (VST-PCA). The model utilizes the video swin transformer to acquire transformation rules, enabling a more accurate capture of the spatiotemporal dependencies inherent in LUCC. Simultaneously, a pre-allocation strategy is introduced in the CA simulation to address the local constraints of neighborhood effects, thereby enhancing the simulation accuracy. Using the Chongqing metropolitan area as the study area, two traditional CA models and two deep learning-based CA models were constructed to validate the performance of the VST-PCA model. Results indicated that the proposed VST-PCA model achieved Kappa and FOM values of 0.8654 and 0.4534, respectively. Compared to other models, Kappa increased by 0.0322–0.1036, and FOM increased by 0.0513–0.1649. This study provides an accurate and effective method for LUCC simulation, offering valuable insights for future research and land management planning.

Spectral feature extraction of rocket exhaust plume using spectral proper orthogonal decomposition

The spectral characterization of flow-field parameters provides a new perspective for understanding the spatiotemporal evolution of unsteady supersonic exhaust plumes and for extracting typical structures. In this study, a large-eddy simulation is performed to calculate the three-dimensional unsteady supersonic plume flow field of rocket engines, and a spectral proper orthogonal decomposition (SPOD) method with a spatiotemporal separation is established. This approach is used to extract the coherent structural features of the unsteady exhaust plume flow field and analyze the mode space structure at different frequencies. The three-dimensional reconstruction and denoising of the exhaust plume flow-field parameters can be achieved via the frequency- and time-domain reconstructions of the SPOD algorithm and oblique projection method, respectively. The ground rocket exhaust plume of ballistic evaluation motor-II is analyzed. The results indicate that the SPOD method can effectively extract the single-frequency mode structure of the reactive supersonic flow field, and that low-order behavior appears in the m = 0 and m = 1 azimuth modes. The potential core exhibits a high-frequency wave-packet structure that is affected by shock waves and shear layers. Time-domain reconstruction based on the oblique projection method facilitates the capture of the dynamic characteristics of the flow field. For the first-order SPOD mode, the frequency- and time-domain reconstruction errors are 3.3% and 1.5%, respectively. The frequency-domain reconstruction method exhibits a 4% improvement in denoising ability compared to low-pass filters. This study provides a novel method for the spectral characterization and spatiotemporal feature extraction of supersonic exhaust plume flow fields.

A two-stage transformer based network for motor imagery classification

Brain-computer interfaces (BCIs) are used to understand brain functioning and develop therapies for neurological and neurodegenerative disorders. Therefore, BCIs are crucial in rehabilitating motor dysfunction and advancing motor imagery applications. For motor imagery, electroencephalogram (EEG) signals are used to classify the subject's intention of moving a body part without actually moving it. This paper presents a two-stage transformer-based architecture that employs handcrafted features and deep learning techniques to enhance the classification performance on benchmarked EEG signals. Stage-1 is built on parallel convolution based EEGNet, multi-head attention, and separable temporal convolution networks for spatiotemporal feature extraction. Further, for enhanced classification, in stage-2, additional features and embeddings extracted from stage-1 are used to train TabNet. In addition, a novel channel cluster swapping data augmentation technique is also developed to handle the issue of limited samples for training deep learning architectures. The developed two-stage architecture offered an average classification accuracy of 88.5 % and 88.3 % on the BCI Competition IV-2a and IV-2b datasets, respectively, which is approximately 3.0 % superior over similar recent reported works.

Vessel trajectory prediction based on spatio-temporal graph convolutional network for complex and crowded sea areas

In order to improve the navigation ability of vessels and ensure the safety of maritime traffic, vessel trajectory prediction plays a crucial role in the intelligent navigation and collision avoidance. Especially for complex and crowded waters, autonomous vessels must have high situational awareness to detect other vessels, and predict their future trajectories and assess collision risks. In this work, a deep attention-aware spatio-temporal graph convolutional network based on AIS data (DAA-SGCN) is proposed to predict the future trajectories of vessels. It mainly includes three modules: motion information encoding of vessel trajectories, the spatio-temporal feature extraction module and the trajectory prediction module. The LSTM is used to extract the motion features of vessels, the spatio-temporal graph is constructed based on deep attention mechanism. The spatial social interaction features are extracted by ST-GCN, and the temporal correlation are extracted by RT-CNN, to obtain the high-level spatio-temporal features of vessel trajectories. Then, the feature embedding is fed into the trajectory prediction module to predict the future trajectories of vessels. On the basis of a large number of experiments, the prediction performance of the DAA-SGCN compared to the optimal baseline model is improved by about 74% and 69% in ADE and FDE metrics.

Classification Algorithm for fNIRS-based Brain Signals Using Convolutional Neural Network with Spatiotemporal Feature Extraction Mechanism

Brain Computer Interface (BCI) is a highly promising human–computer interaction method that can utilize brain signals to control external devices. BCI based on functional near-infrared spectroscopy (fNIRS) is considered a relatively new and promising paradigm. fNIRS is a technique of measuring functional changes in cerebral hemodynamics. It detects changes in the hemodynamic activity of the cerebral cortex by measuring oxyhemoglobin and deoxyhemoglobin (HbR) concentrations and inversely predicts the neural activity of the brain. At the present time, Deep learning (DL) methods have not been widely used in fNIRS decoding, and there are fewer studies considering both spatial and temporal dimensions for fNIRS classification. To solve these problems, we proposed an end-to-end hybrid neural network for feature extraction of fNIRS. The method utilizes a spatial–temporal convolutional layer for automatic extraction of temporally valid information and uses a spatial attention mechanism to extract spatially localized information. A temporal convolutional network (TCN) is used to further utilize the temporal information of fNIRS before the fully connected layer. We validated our approach on a publicly available dataset including 29 subjects, including left-hand and right-hand motor imagery (MI), mental arithmetic (MA), and a baseline task. The results show that the method has few training parameters and high accuracy, providing a meaningful reference for BCI development.

Coupling video vision transformer (ViVit) into land change simulation: a comparison with three-dimensional convolutional neural network (3DCNN)

ABSTRACT To enhance land use/cover change (LUCC) simulation accuracy, we introduced ViViT-ANN-CA, blending video vision transformer’s spatio-temporal features extraction ability, artificial neural network‘s (ANN) non-linearity computing ability, and CA’s spatial computing. Compared to 3DCNN-ANN-CA, ViViT-ANN-CA showed higher accuracy in simulating water bodies and vegetation, with overall improvements in Hailing District and Wuxi City. ViViT demonstrates comparable spatio-temporal feature extraction ability to three-dimensional convolutional neural network (3DCNN), promising for future ynamic LUCC simulations.

Analog spatiotemporal feature extraction for cognitive radio-frequency sensing with integrated photonics

Analog feature extraction (AFE) is an appealing strategy for low-latency and efficient cognitive sensing systems since key features are much sparser than the Nyquist-sampled data. However, applying AFE to broadband radio-frequency (RF) scenarios is challenging due to the bandwidth and programmability bottlenecks of analog electronic circuitry. Here, we introduce a photonics-based scheme that extracts spatiotemporal features from broadband RF signals in the analog domain. The feature extractor structure inspired by convolutional neural networks is implemented on integrated photonic circuits to process RF signals from multiple antennas, extracting valid features from both temporal and spatial dimensions. Because of the tunability of the photonic devices, the photonic spatiotemporal feature extractor is trainable, which enhances the validity of the extracted features. Moreover, a digital-analog-hybrid transfer learning method is proposed for the effective and low-cost training of the photonic feature extractor. To validate our scheme, we demonstrate a radar target recognition task with a 4-GHz instantaneous bandwidth. Experimental results indicate that the photonic analog feature extractor tackles broadband RF signals and reduces the sampling rate of analog-to-digital converters to 1/4 of the Nyquist sampling while maintaining a high target recognition accuracy of 97.5%. Our scheme offers a promising path for exploiting the AFE strategy in the realm of cognitive RF sensing, with the potential to contribute to the efficient signal processing involved in applications such as autonomous driving, robotics, and smart factories.

Tropical cyclone trajectory based on satellite remote sensing prediction and time attention mechanism ConvLSTM model

The accurate and timely prediction of tropical cyclones is of paramount importance in mitigating the impact of these catastrophic meteorological events. Presently, methods for predicting tropical cyclones based on satellite remote sensing images encounter notable challenges, including the inadequate extraction of three-dimensional spatial features and limitations in long-term forecasting. As a response to these challenges, this study introduces the Temporal Attention Mechanism ConvLSTM (TAM-CL) model, designed to conduct thorough spatiotemporal feature extraction on three-dimensional atmospheric reanalysis data of tropical cyclones. By leveraging ConvLSTM with three-dimensional convolution kernels, our model enhances the extraction of three-dimensional spatiotemporal features. Furthermore, an attention mechanism is integrated to bolster long-term prediction accuracy by emphasizing crucial temporal nodes. In the evaluation of tropical cyclone track and intensity forecasts across 24, 48, and 72 h, TAM-CL demonstrates a notable reduction in prediction errors, thereby underscoring its efficacy in forecasting both cyclone tracks and intensities. This contributes to an effective exploration of the application of deep networks in conjunction with atmospheric reanalysis data.